Skip to content

Google: Add retry-safe batch_id_prefix to DataprocCreateBatchOperator#69855

Open
amirmor1 wants to merge 2 commits into
apache:mainfrom
amirmor1:retry-with-unique-batch-id
Open

Google: Add retry-safe batch_id_prefix to DataprocCreateBatchOperator#69855
amirmor1 wants to merge 2 commits into
apache:mainfrom
amirmor1:retry-with-unique-batch-id

Conversation

@amirmor1

Copy link
Copy Markdown
Contributor

Add an opt-in batch_id_prefix to DataprocCreateBatchOperator so task retries can submit a new Dataproc batch with a unique ID instead of reusing a static batch_id.

Why

When a task retries with the same batch_id, Dataproc can return ALREADY_EXISTS and attach to the old batch. This change provides a built-in retry-safe path without changing existing explicit batch_id behavior.

What changes

  • add optional batch_id_prefix parameter
  • generate <batch_id_prefix>-<unique_suffix> per task attempt
  • keep batch_id and batch_id_prefix mutually exclusive
  • keep explicit batch_id behavior unchanged
  • update Dataproc operator docs
  • add unit tests for:
    • mutual exclusivity
    • prefix-based ID generation
    • execute flow with prefix
    • backward compatibility for explicit batch_id

Test plan

  • uv run ruff format providers/google/src/airflow/providers/google/cloud/operators/dataproc.py providers/google/tests/unit/google/cloud/operators/test_dataproc.py
  • uv run ruff check --fix providers/google/src/airflow/providers/google/cloud/operators/dataproc.py providers/google/tests/unit/google/cloud/operators/test_dataproc.py
  • uv run --project providers/google pytest providers/google/tests/unit/google/cloud/operators/test_dataproc.py::TestDataprocCreateBatchOperator::test_execute providers/google/tests/unit/google/cloud/operators/test_dataproc.py::TestDataprocCreateBatchOperator::test_execute_with_batch_id_prefix providers/google/tests/unit/google/cloud/operators/test_dataproc.py::TestDataprocCreateBatchOperator::test_build_unique_batch_id_from_prefix_preserves_prefix_verbatim providers/google/tests/unit/google/cloud/operators/test_dataproc.py::TestDataprocCreateBatchOperator::test_batch_id_and_batch_id_prefix_are_mutually_exclusive -xvs

@amirmor1 amirmor1 requested a review from shahar1 as a code owner July 14, 2026 07:19
@boring-cyborg boring-cyborg Bot added area:providers kind:documentation provider:google Google (including GCP) related issues labels Jul 14, 2026
@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers kind:documentation provider:google Google (including GCP) related issues ready for maintainer review Set after triaging when all criteria pass.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants